-
Notifications
You must be signed in to change notification settings - Fork 320
[OCX-87] Feat/Export #2569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[OCX-87] Feat/Export #2569
Conversation
- Add new export endpoint in `views_v2.py` to export Safe transaction history
- Implement comprehensive SQL query to aggregate multisig transactions, module transactions, and token transfers
- Create `TransactionExportSerializer` for structured export data format
- Add transaction export service in transaction_service.py with deduplication logic
- Include support for ERC20, ERC721, and native ETH transfers
- Add URL routing for `/api/v2/safes/{address}/export/` endpoint
- Add comprehensive test coverage for export functionality
| parsed_execution_date_gte = None | ||
| parsed_execution_date_lte = None | ||
|
|
||
| if execution_date_gte: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you can use django.utils.dateparse.parse_datetime?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wrote a serializer to parse this values. ae8a43e
| }, | ||
| ) | ||
|
|
||
| if execution_date_lte: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above. #2569 (comment)
| params = [] | ||
|
|
||
| if execution_date_gte: | ||
| where_conditions.append("execution_date >= %s") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you should assert that execution_date_gte is really a datetime, as it can lead to sql injection if called with a string from other part of the code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done :)
| where_clause = " AND ".join(where_conditions) if where_conditions else "1=1" | ||
|
|
||
| # Main query that unions all transaction types with their transfers | ||
| main_query = f""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on our experience with all-transactions endpoint, this query will be really slow to the point it will not end for some big Safes. The relevant txs table needs to be used to filter, and then join the required tables to get the information (take a look at the all-transactions endpoint)
The query needs to be tested in our staging database
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Completely agree with you..
As I added tests would be easier try new approach of this query, for now I would say leave as is, anyway I did some performance improvements as remove unnecessary JOINS in some queries.
@Uxio0 I think the count should be exactly the same than count the erc20, erc721 and the intenal transactions, maybe would be simpler than the current approach, what do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense, yes
| # Add '0x' prefix to hex strings and convert addresses to checksum format (except for null values) | ||
| if row_dict["safe_address"]: | ||
| address = "0x" + row_dict["safe_address"] | ||
| row_dict["safe_address"] = Web3.to_checksum_address(address) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Web3.to_checksum_address is really slow, better use safe_eth.eth.utils.fast_to_checksum_address
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done :)
Add tests for moduletransactions sending ether
| ) | ||
| ) combined | ||
| ) | ||
| SELECT COUNT(DISTINCT (execution_date, transaction_hash)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Uxio0 @gjeanmart I think this could give as inconsistent count, same transaction hash in the same date could contains several ERC20 transfers, would makes sense use trace_address and log_index to differenciate it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, well spotted
| try: | ||
| # Handle different ISO format variations | ||
| date_str = execution_date_gte.replace("Z", "+00:00") | ||
| # Fix format issue where timezone offset has space instead of + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Time zone offset is necessary on ISO 8601 format. https://docs.digi.com/resources/documentation/digidocs/90001488-13/reference/r_iso_8601_date_format.htm
Basically the + symbol was omitted because the url was not encoded correctly in the request to scape the "+".
ae8a43e#diff-24713fcb3b2d8297aa5713f19f963ed46b80f4ee025980b00a375d7d59e083dcR2688
| _trace_address=F("trace_address"), | ||
| ).values("transaction_hash", "_log_index", "_trace_address") | ||
|
|
||
| total_count = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The previous count was not working well returning more or less items than the real count, so just counting the transfers would be enough.
Regarding performance analyze this is the result of the new query:
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------
Append (cost=0.12..32.85 rows=3 width=96) (actual time=0.015..0.016 rows=0 loops=1)
-> Index Scan using history_internal_transfer_from on history_internaltx (cost=0.12..8.15 rows=1 width=580) (actual time=0.004..0.004 rows=0 loops=1)
Index Cond: ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone)
Filter: (("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) OR (_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea))
-> Subquery Scan on "*SELECT* 2" (cost=8.32..12.35 rows=1 width=96) (actual time=0.009..0.010 rows=0 loops=1)
-> Bitmap Heap Scan on history_erc20transfer (cost=8.32..12.33 rows=1 width=68) (actual time=0.009..0.009 rows=0 loops=1)
Recheck Cond: ((("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone)) OR ((_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with t
ime zone)))
-> BitmapOr (cost=8.32..8.32 rows=1 width=0) (actual time=0.008..0.009 rows=0 loops=1)
-> Bitmap Index Scan on history_erc_to_f32154_idx (cost=0.00..4.16 rows=1 width=0) (actual time=0.005..0.005 rows=0 loops=1)
Index Cond: (("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone))
-> Bitmap Index Scan on history_erc__from_64986c_idx (cost=0.00..4.16 rows=1 width=0) (actual time=0.003..0.003 rows=0 loops=1)
Index Cond: ((_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone))
-> Subquery Scan on "*SELECT* 3" (cost=8.32..12.35 rows=1 width=96) (actual time=0.002..0.002 rows=0 loops=1)
-> Bitmap Heap Scan on history_erc721transfer (cost=8.32..12.33 rows=1 width=68) (actual time=0.002..0.002 rows=0 loops=1)
Recheck Cond: ((("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone)) OR ((_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with t
ime zone)))
-> BitmapOr (cost=8.32..8.32 rows=1 width=0) (actual time=0.001..0.002 rows=0 loops=1)
-> Bitmap Index Scan on history_erc_to_02d4ab_idx (cost=0.00..4.16 rows=1 width=0) (actual time=0.000..0.000 rows=0 loops=1)
Index Cond: (("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone))
-> Bitmap Index Scan on history_erc__from_72fb41_idx (cost=0.00..4.16 rows=1 width=0) (actual time=0.000..0.000 rows=0 loops=1)
Index Cond: ((_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) AND ("timestamp" >= '2025-07-18 09:58:33.305547+00'::timestamp with time zone))
Planning Time: 3.412 ms
Execution Time: 0.068 ms
Just for comparation, this is the previous query plan.
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Append (cost=0.58..153.86 rows=12 width=40) (actual time=0.088..0.093 rows=0 loops=1)
-> Nested Loop (cost=0.58..13.40 rows=2 width=41) (actual time=0.019..0.020 rows=0 loops=1)
Join Filter: (mt.ethereum_tx_id = erc20.ethereum_tx_id)
-> Nested Loop Left Join (cost=0.44..12.89 rows=1 width=73) (actual time=0.018..0.019 rows=0 loops=1)
-> Nested Loop (cost=0.29..12.42 rows=1 width=69) (actual time=0.018..0.019 rows=0 loops=1)
-> Index Scan using history_multisigtransaction_safe_ba8bae68 on history_multisigtransaction mt (cost=0.14..4.16 rows=1 width=33) (actual time=0.018..0.018 rows=0 loops=1)
Index Cond: (safe = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea)
-> Index Scan using history_ethereumtx_pkey on history_ethereumtx et (cost=0.14..8.16 rows=1 width=36) (never executed)
Index Cond: (tx_hash = mt.ethereum_tx_id)
-> Index Scan using history_ethereumblock_pkey on history_ethereumblock eb (cost=0.15..0.46 rows=1 width=12) (never executed)
Index Cond: (number = et.block_id)
-> Index Only Scan using unique_erc20_transfer_index on history_erc20transfer erc20 (cost=0.15..0.48 rows=2 width=32) (never executed)
Index Cond: (ethereum_tx_id = et.tx_hash)
Heap Fetches: 0
-> Nested Loop (cost=0.58..13.37 rows=2 width=41) (actual time=0.008..0.008 rows=0 loops=1)
Join Filter: (mt_1.ethereum_tx_id = erc721.ethereum_tx_id)
-> Nested Loop Left Join (cost=0.44..12.89 rows=1 width=73) (actual time=0.008..0.008 rows=0 loops=1)
-> Nested Loop (cost=0.29..12.42 rows=1 width=69) (actual time=0.008..0.008 rows=0 loops=1)
-> Index Scan using history_multisigtransaction_safe_ba8bae68 on history_multisigtransaction mt_1 (cost=0.14..4.16 rows=1 width=33) (actual time=0.008..0.008 rows=0 loops=1)
Index Cond: (safe = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea)
-> Index Scan using history_ethereumtx_pkey on history_ethereumtx et_1 (cost=0.14..8.16 rows=1 width=36) (never executed)
Index Cond: (tx_hash = mt_1.ethereum_tx_id)
-> Index Scan using history_ethereumblock_pkey on history_ethereumblock eb_1 (cost=0.15..0.46 rows=1 width=12) (never executed)
Index Cond: (number = et_1.block_id)
-> Index Only Scan using unique_erc721_transfer_index on history_erc721transfer erc721 (cost=0.15..0.46 rows=2 width=32) (never executed)
Index Cond: (ethereum_tx_id = et_1.tx_hash)
Heap Fetches: 0
-> Nested Loop Anti Join (cost=0.73..13.70 rows=1 width=41) (actual time=0.005..0.006 rows=0 loops=1)
-> Nested Loop Anti Join (cost=0.58..13.26 rows=1 width=73) (actual time=0.005..0.006 rows=0 loops=1)
-> Nested Loop Left Join (cost=0.44..12.80 rows=1 width=73) (actual time=0.005..0.005 rows=0 loops=1)
-> Index Scan using history_multisigtransaction_safe_ba8bae68 on history_multisigtransaction mt_2 (cost=0.14..4.16 rows=1 width=33) (actual time=0.005..0.005 rows=0 loops=1)
Index Cond: (safe = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea)
-> Nested Loop Left Join (cost=0.29..8.63 rows=1 width=40) (never executed)
-> Index Scan using history_ethereumtx_pkey on history_ethereumtx et_2 (cost=0.14..8.16 rows=1 width=36) (never executed)
Index Cond: (tx_hash = mt_2.ethereum_tx_id)
-> Index Scan using history_ethereumblock_pkey on history_ethereumblock eb_2 (cost=0.15..0.46 rows=1 width=12) (never executed)
Index Cond: (number = et_2.block_id)
-> Index Only Scan using unique_erc20_transfer_index on history_erc20transfer erc20_1 (cost=0.15..0.48 rows=2 width=32) (never executed)
Index Cond: (ethereum_tx_id = et_2.tx_hash)
Heap Fetches: 0
-> Index Only Scan using unique_erc721_transfer_index on history_erc721transfer erc721_1 (cost=0.15..0.46 rows=2 width=32) (never executed)
Index Cond: (ethereum_tx_id = et_2.tx_hash)
Heap Fetches: 0
-> Hash Join (cost=9.53..20.16 rows=2 width=40) (actual time=0.005..0.005 rows=0 loops=1)
Hash Cond: (itx.id = modtx.internal_tx_id)
-> Seq Scan on history_internaltx itx (cost=0.00..10.50 rows=50 width=48) (actual time=0.004..0.004 rows=0 loops=1)
-> Hash (cost=9.50..9.50 rows=2 width=8) (never executed)
-> Bitmap Heap Scan on history_moduletransaction modtx (cost=4.16..9.50 rows=2 width=8) (never executed)
Recheck Cond: (safe = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea)
-> Bitmap Index Scan on history_moduletransaction_safe (cost=0.00..4.16 rows=2 width=0) (never executed)
Index Cond: (safe = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea)
-> Nested Loop Anti Join (cost=15.74..34.43 rows=2 width=40) (actual time=0.020..0.021 rows=0 loops=1)
-> Hash Right Anti Join (cost=15.45..22.37 rows=3 width=40) (actual time=0.020..0.021 rows=0 loops=1)
Hash Cond: (mt_3.ethereum_tx_id = erc20_2.ethereum_tx_id)
-> Seq Scan on history_multisigtransaction mt_3 (cost=0.00..6.39 rows=139 width=33) (never executed)
-> Hash (cost=15.40..15.40 rows=4 width=40) (actual time=0.007..0.007 rows=0 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 8kB
-> Seq Scan on history_erc20transfer erc20_2 (cost=0.00..15.40 rows=4 width=40) (actual time=0.007..0.007 rows=0 loops=1)
Filter: (("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) OR (_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea))
Rows Removed by Filter: 4
-> Nested Loop (cost=0.29..6.21 rows=7 width=32) (never executed)
-> Index Scan using unique_internal_tx_trace_address on history_internaltx itx_1 (cost=0.14..5.16 rows=1 width=40) (never executed)
Index Cond: (ethereum_tx_id = erc20_2.ethereum_tx_id)
-> Index Only Scan using history_moduletransaction_pkey on history_moduletransaction modtx_1 (cost=0.15..1.05 rows=1 width=8) (never executed)
Index Cond: (internal_tx_id = itx_1.id)
Heap Fetches: 0
-> Nested Loop Anti Join (cost=15.74..34.43 rows=2 width=40) (actual time=0.013..0.013 rows=0 loops=1)
-> Hash Right Anti Join (cost=15.45..22.37 rows=3 width=40) (actual time=0.012..0.013 rows=0 loops=1)
Hash Cond: (mt_4.ethereum_tx_id = erc721_2.ethereum_tx_id)
-> Seq Scan on history_multisigtransaction mt_4 (cost=0.00..6.39 rows=139 width=33) (never executed)
-> Hash (cost=15.40..15.40 rows=4 width=40) (actual time=0.001..0.001 rows=0 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 8kB
-> Seq Scan on history_erc721transfer erc721_2 (cost=0.00..15.40 rows=4 width=40) (actual time=0.001..0.001 rows=0 loops=1)
Filter: (("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) OR (_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea))
-> Nested Loop (cost=0.29..6.21 rows=7 width=32) (never executed)
-> Index Scan using unique_internal_tx_trace_address on history_internaltx itx_2 (cost=0.14..5.16 rows=1 width=40) (never executed)
Index Cond: (ethereum_tx_id = erc721_2.ethereum_tx_id)
-> Index Only Scan using history_moduletransaction_pkey on history_moduletransaction modtx_2 (cost=0.15..1.05 rows=1 width=8) (never executed)
Index Cond: (internal_tx_id = itx_2.id)
Heap Fetches: 0
-> Nested Loop Anti Join (cost=8.44..24.30 rows=1 width=40) (actual time=0.018..0.018 rows=0 loops=1)
-> Hash Right Anti Join (cost=8.16..15.08 rows=1 width=40) (actual time=0.018..0.018 rows=0 loops=1)
Hash Cond: (mt_5.ethereum_tx_id = itx_3.ethereum_tx_id)
-> Seq Scan on history_multisigtransaction mt_5 (cost=0.00..6.39 rows=139 width=33) (never executed)
-> Hash (cost=8.14..8.14 rows=1 width=40) (actual time=0.005..0.005 rows=0 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 8kB
-> Index Scan using history_internal_transfer_from on history_internaltx itx_3 (cost=0.12..8.14 rows=1 width=40) (actual time=0.005..0.005 rows=0 loops=1)
Filter: (("to" = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea) OR (_from = '\x4b1c677b856d27d7a2cbefdda993fc7e6f83c502'::bytea))
-> Nested Loop (cost=0.29..9.21 rows=7 width=32) (never executed)
-> Index Scan using unique_internal_tx_trace_address on history_internaltx itx2 (cost=0.14..8.16 rows=1 width=40) (never executed)
Index Cond: (ethereum_tx_id = itx_3.ethereum_tx_id)
-> Index Only Scan using history_moduletransaction_pkey on history_moduletransaction modtx_3 (cost=0.15..1.05 rows=1 width=8) (never executed)
Index Cond: (internal_tx_id = itx2.id)
Heap Fetches: 0
Planning Time: 6.984 ms
Execution Time: 0.455 ms
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice
| safeTxHash = Sha3HashField(source="safe_tx_hash", allow_null=True) | ||
| method = serializers.CharField(allow_null=True) | ||
| contractAddress = EthereumAddressField(source="contract_address", allow_null=True) | ||
| isExecuted = serializers.BooleanField(source="is_executed") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it always be true?
| ), | ||
| path( | ||
| "safes/<str:address>/export/", | ||
| views_v2.SafeExportView.as_view(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to be stay aligned, should it go in V1 or V2?
So far we only have in v2, endpoints that were replaced from v1
| where_clause = " AND ".join(where_conditions) if where_conditions else "1=1" | ||
|
|
||
| # Main query that unions all transaction types with their transfers | ||
| main_query = f""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find it strange to see SQL in the service layer, would it make sense to move it to models?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is also the same for the all transaction but using the ORM.
I agree with you but I would try first to migrate the query to the ORM and check the performance before do a refactor for this.
Make sure these boxes are checked! 📦✅
./run_tests.shpre-commit run -asetup_service.py, provide a link to yoursafe-deployments PR and check network name
exists in safe-eth-py
What was wrong? 👾
Closes OCX-87
How was it fixed? 🎯
Summary
This PR introduces a new export endpoint that allows users to retrieve a comprehensive transaction history for Safe wallets in a unified format. The endpoint aggregates all transaction types (multisig, module, and standalone transfers) and provides detailed information about each transaction including token transfers.
What This PR Does
/api/v2/safes/{address}/export/endpointWhy This Change Is Needed
Users need a single endpoint to export their complete Safe transaction history for:
Previously, users had to make multiple API calls to different endpoints to get complete transaction data, making it difficult to get a unified view of their Safe's activity.
Implementation Details
New SQL Query Features
API Design
GET /api/v2/safes/{address}/export/Data Structure
Each export record includes:
Files Changed
safe_transaction_service/history/views_v2.py- New export endpointsafe_transaction_service/history/serializers.py- Export serializersafe_transaction_service/history/services/transaction_service.py- Export service logicsafe_transaction_service/history/urls_v2.py- URL routingsafe_transaction_service/history/tests/test_views_v2.py- Test coverageTesting
How to Test
Technical Notes
Breaking Changes
None - this is a new endpoint addition.
Migration Required
None - no database schema changes.